Information processing apparatus, display control method, and storage medium

ABSTRACT

An information processing apparatus includes a reception unit configured to receive a designation of a virtual viewpoint relating to a virtual viewpoint image, and a display control unit configured to display a virtual viewpoint image which is generated based on images obtained by shooting an imaging target area from a plurality of directions using a plurality of imaging apparatuses and corresponds to a virtual viewpoint according to the designation received by the reception unit, and information, which specifies a target object included in a range of the field of view of a virtual viewpoint different from the virtual viewpoint according to the designation, on a display unit, wherein the target object is in the imaging target area.

BACKGROUND Field of the Disclosure

The present disclosure relates to a virtual viewpoint image.

Description of the Related Art

There is a technique for synchronously capturing images on amulti-viewpoint basis by using a plurality of imaging apparatusesinstalled at different positions and, by using a plurality of imagesobtained in this image capturing, generating not only images captured atthe installation positions of the imaging apparatuses but also a virtualviewpoint image which can be obtained by arbitrarily changing theviewpoint.

The virtual viewpoint image is generated by an image processing unitsuch as a server, which consolidates images captured by a plurality ofcameras, generates a three-dimensional model, and performs processingsuch as rendering on the model. The generated virtual viewpoint image isthen transmitted to a user terminal to be viewed.

Japanese Patent Application Laid-Open No. 2014-215828 discusses atechnique for generating and displaying the virtual viewpoint imagecorresponding to an arbitrary viewpoint setting by using images in anidentical range captured by a plurality of imaging apparatuses arrangedto surround the identical range.

For example, if a virtual viewpoint image according to a user-setviewpoint is generated from captured images of a sporting event, a usercan watch a game from a desired viewpoint. However, a user unfamiliarwith virtual viewpoint setting may hardly know a viewpoint at which heor she should watch a viewing target (such as a sporting event) in orderto generate a realistic virtual viewpoint image. In addition, a userunfamiliar with virtual viewpoint setting is unable to recognize theposition of a virtual viewpoint which enables shooting of a remarkableevent or object viewed by many other users. Therefore, such a user maypossibly miss the remarkable event or object. A viewing target accordingto a virtual viewpoint image is not limited to a sporting event, and asimilar problem may occur in a concert and other events.

SUMMARY

According to an aspect of the present disclosure, an informationprocessing apparatus includes a reception unit configured to receive adesignation of a virtual viewpoint relating to a virtual viewpointimage, and a display control unit configured to display a virtualviewpoint image which is generated based on images obtained by shootingan imaging target area from a plurality of directions using a pluralityof imaging apparatuses and corresponds to a virtual viewpoint accordingto the designation received by the reception unit, and information,which specifies a target object included in a range of the field of viewof a virtual viewpoint different from the virtual viewpoint according tothe designation, on a display unit, wherein the target object is in theimaging target area.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration of an image processing system.

FIG. 2 illustrates an example of virtual camera visualizationinformation.

FIG. 3 illustrates another example of the virtual camera visualizationinformation.

FIG. 4 is a flowchart illustrating a flow of processing for generatingthe virtual camera visualization information.

FIGS. 5A and 5B illustrate other examples of the virtual cameravisualization information.

FIGS. 6A to 6D illustrate examples of display screens of a userterminal.

FIG. 7 is a flowchart illustrating a flow of recommendation processing.

FIG. 8 is another flowchart illustrating a flow of processing forgenerating the virtual camera visualization information.

FIG. 9 illustrates a still another example of the virtual cameravisualization information.

FIG. 10 illustrates a still another example of the virtual cameravisualization information.

FIG. 11 illustrates a hardware configuration of each apparatus of theimage processing system.

FIG. 12 is a flowchart illustrating flow of processing of an imageprocessing apparatus.

FIG. 13 is a flowchart illustrating a flow of processing of the userterminal.

DESCRIPTION OF THE EMBODIMENTS

An exemplary embodiment of the present disclosure will be described indetail below with reference to the accompanying drawings. However, thepresent disclosure is not limited to the following exemplary embodimentbut can be modified in diverse ways within the ambit of the appendedclaims.

(Image Processing System)

An exemplary embodiment will be described below with reference to theaccompanying drawings. FIG. 1 schematically illustrates an example of animage processing system according to the present exemplary embodiment.An image processing system 100 illustrated in FIG. 1 is a system forgenerating a virtual viewpoint image by using images captured by aplurality of cameras installed on a field where a game is actuallyperformed, and on a stadium or concert hall holding spectators' seats. Avirtual viewpoint image is an image generated by using a plurality ofimages captured at a plurality of viewpoints, and represents a shootingresult of a virtual image at an arbitrary viewpoint or arbitraryline-of-sight direction. The image processing system 100 includes animage processing apparatus 3, user terminals 4 a to 4 z, and a storagedevice 5.

The storage device 5 is an apparatus for storing various informationincluding a plurality of images and imaging targets shot by a pluralityof cameras installed in the stadium, and information about the stadiumand field which are spaces and also to be shot. The image processingapparatus 3 is an information processing apparatus for generatinginformation for visualizing viewpoints according to virtual viewpointimages set by a plurality of users and generating virtual viewpointimages. The image processing apparatus 3 generates a virtual viewpointimage representing the shooting result of the virtual image based on theposition of a determined viewpoint. The image processing apparatus 3 mayalso generate a virtual viewpoint image representing an image shootingresult when a virtual camera is oriented in a determined line-of-sightdirection. The image processing apparatus 3 generates a virtualviewpoint image by using a plurality of images captured by a pluralityof cameras and accumulated in the storage device 5. The image processingapparatus 3 transmits the generated virtual viewpoint image to any ofthe user terminals 4 a to 4 z.

Each of the user terminals 4 a to 4 z displays the virtual viewpointimage received from the image processing apparatus 3 on the displayscreen provided on the user terminals 4 a to 4 z. The user terminals 4 ato 4 z also receives, for example, an instruction for setting and movingthe position of a viewpoint related to a virtual viewpoint image, aninstruction for setting and changing the line-of-sight direction relatedto a virtual viewpoint image, and an instruction for switching theviewpoint based on a user's input. Then, the user terminals 4 a to 4 ztransmit as virtual viewpoint information a transmission signalindicating the contents of the instruction to the image processingapparatus 3. Unless specifically noted, 26 user terminals (the userterminals 4 a to 4 z) are collectively referred to as a user terminal 4,and components of the user terminal 4 will be described below withoutdistinction between the user terminals 4 a to 4 z. Although the numberof user terminals is 26, the number is only an example and is notlimited thereto. Unless otherwise noted, in the present exemplaryembodiment, the word image indicates concepts of both a moving image anda still image. More specifically, the image processing system 100according to the present exemplary embodiment is capable of processingboth still images and moving images.

An example of a hardware configuration of each apparatus (the imageprocessing apparatus 3, the user terminals 4 a to 4 z, and the storagedevice 5) in the image processing system 100 will be described belowwith reference to FIG. 11. An apparatus 1200 includes a centralprocessing unit (CPU) 1201, a read only memory (ROM) 1202, a randomaccess memory (RAM) 1203, an auxiliary storage device 1204, a displayunit 1205, an operation unit 1206, a communication unit 1207, and a bus1208.

The CPU 1201 controls the entire apparatus 1200 by using computerprograms and data stored in the ROM 1202 and RAM 1203. The ROM 1202stores programs and parameters which do not need to be changed. The RAM1203 temporarily stores programs and data supplied from the auxiliarystorage device 1204 and data supplied from the outside via thecommunication unit 1207. The auxiliary storage device 1204 including ahard disk drive stores content data such as still images and movingimages.

The display unit 1205 including a liquid crystal display (LCD) displaysa graphical user interface (GUI) used by the user to operate theapparatus 1200. The operation unit 1206 including a keyboard and a mousereceives user instructions and outputs various instructions to the CPU1201. The communication unit 1207 communicates with an externalapparatus. For example, when the apparatus 1200 is connected via wiredcommunication with an external apparatus, a local area network (LAN)cable is connected to the communication unit 1207. When the apparatus1200 has a function of wirelessly communicating with an externalapparatus, the communication unit 1207 has an antenna. The bus 1208connects the above-described components of the apparatus 1200 andtransmits information therebetween.

Although, in the present exemplary embodiment, the display unit 1205 andthe operation unit 1206 are included in the apparatus 1200, theapparatus 1200 does not need to include at least one of the display unit1205 and the operation unit 1206. In addition, at least one of thedisplay unit 1205 and the operation unit 1206 may exist outside theapparatus 1200 as a separate apparatus, and the CPU 1201 may operate asa display control unit for controlling the display unit 1205 and as anoperation control unit for controlling the operation unit 1206.

The CPU 1201 may be composed of a single CPU or a plurality of CPUs, ormay be a multi-core CPU. Instead of the CPU 1201 or together with theCPU 1201, the apparatus 1200 may include a hardware component such as anapplication specific integrated circuit (ASIC), field programmable gatearray (FPGA), or graphics processing unit (GPU). In this case, ahardware component such as an ASIC, FPGA, or GPU may perform part or allof processing to be executed by the CPU 1201. Part of the processing ofthe apparatus 1200 may be implemented by hardware, and another part ofthe processing may be implemented through software processing carriedout by the CPU 1201. Not all of the apparatuses of the image processingsystem 100 need to have a configuration similar to the above. Eachindividual apparatus may lack a part of the configuration or may haveother configurations.

The functional configuration of each apparatus of the image processingsystem 100 will be described below with reference to FIG. 1. Thefunctional configuration of each device illustrated in FIG. 1 isimplemented when the CPU 1201 of each device executes a computer programstored in the ROM 1202 or RAM 1203 to control information calculationsand each hardware component. Part or all of each configurationillustrated in FIG. 1 may be implemented by a dedicated hardwarecomponent. The dedicated hardware component is, for example, an ASIC,FPGA, or GPU.

A configuration of the storage device 5 will be described below. Thestorage device 5 includes an image storage unit 51 and a subjectinformation storage unit 52. The image storage unit 51 stores aplurality of images synchronously captured by a plurality of camerasinstalled surrounding a game field. The image storage unit 51 may storecaptured images having undergone foreground and background separationprocessing. In the foreground background separation processing, the areaof a predetermined object (hereinafter referred to as a foregroundobject) as a ball or a specific person such as a player may be extractedas a foreground area from an image, and the remaining image (the imageother than the foreground area extracted) may be a background area. Theimage indicating the foreground area is also called a foreground image,and the image indicating the foreground area is also called a backgroundimage. The subject information storage unit 52 stores information aboutspaces to be shot. The subject information storage unit 52 storesthree-dimensional model information of an object (background object)projected as a background onto virtual viewpoint image, such as thefield and spectators' seats of the stadium which are an imaging target,and three-dimensional space information of a range where virtualviewpoints can be set. Part or all of the above-described functionalconfigurations of the storage device 5 may be included in the imageprocessing apparatus 3 or the user terminal 4.

A configuration of the image processing apparatus 3 will be describedbelow. The image processing apparatus 3 includes a virtual viewpointimage generation unit 301, an acquisition unit 302, a storage unit 303,a visualization information generation unit 304, a display imagegeneration unit 305, a management unit 306, and a communication unit307. The virtual viewpoint image generation unit 301 generates a virtualviewpoint image based on a plurality of images acquired from the imagestorage unit 51. The virtual viewpoint image generation unit 301 maygenerate a three-dimensional model based on a plurality of imagesacquired from the image storage unit 51 and generate a virtual viewpointimage through model base rendering for mapping a texture from theviewpoint related to a virtual viewpoint image acquired from theacquisition unit 302. Model base rendering utilizes a three-dimensionalshape (model) of a target obtained through a three-dimensional shaperestoration technique such as a volume intersection method andMULTI-VIEW-STEREO (MVS). Model base rendering is a technique forgenerating an image captured by a virtual camera at a viewpointaccording to a virtual viewpoint image by using the generatedthree-dimensional shape. A virtual viewpoint image may be generated byusing other methods such as image base rendering. Image base renderingis a rendering method for generating a virtual viewpoint image based onimages captured at a plurality of viewpoints without performing modeling(a process for generating an object shape by using geometric figures).

The virtual viewpoint image generation unit 301 calculates thecoordinates of the foreground and background objects projected onto avirtual viewpoint image to be generated and performs texture mapping andrendering for the coordinates. The virtual viewpoint image generationunit 301 stores the coordinate values related to the foreground objectas the foreground coordinates and stores the coordinate value related tothe background object as the background coordinates in the storage unit303 (described below).

The acquisition unit 302 acquires the viewpoint position and/orline-of-sight direction related to the virtual viewpoint image to begenerated. In the present exemplary embodiment a virtual camera forcapturing and generating the virtual viewpoint image is used, andsetting of the viewpoint position and/or line-of-sight direction relatedto a virtual viewpoint image to be generated is also called setting ofthe position and orientation of the virtual camera. A continuoustransition of the position and orientation of a virtual camera is calleda virtual camera path. More specifically, the image processing apparatus3 can generate a plurality of the virtual viewpoint images based on theset virtual camera path. The image processing apparatus 3 can alsogenerate as one moving image a plurality of the virtual viewpoint imagesgenerated based on the set virtual camera path.

When the virtual viewpoint information based on a user's operation isacquired from the user terminal 4, the acquisition unit 302 determinesthe position and direction corresponding to the virtual viewpointinformation as the viewpoint position and line-of-sight direction,respectively, related to the virtual viewpoint image to be output. Thevirtual viewpoint information includes at least position information anddirection information.

Based on the contents of a user instruction, the acquisition unit 302acquires the position and orientation of a virtual camera related to atemporally continuous virtual camera path or a still image. Theacquisition unit 302 identifies a virtual camera path by setting virtualcamera parameters. The virtual camera parameters include the positionand/or direction of a virtual camera related to the frame number or timecode. The acquisition unit 302 may set the angle of view, focal length,or zoom magnification of a virtual camera as virtual camera parameters.The acquisition unit 302 refers to the subject space informationacquired from the subject information storage unit 52 and sets a virtualcamera path in the range where virtual viewpoints can be set.

The storage unit 303 accumulates the foreground and the backgroundcoordinates received from the virtual viewpoint image generation unit301 and the virtual camera path parameters calculated by the acquisitionunit 302 in association with user information acquired from themanagement unit 306 (described below).

To visualize information about virtual cameras set by a plurality ofusers accumulated in the storage unit 303 so that the user cancomprehensively and intuitively recognize and grasp the information, thevisualization information generation unit 304 generates graphic data ortext data. Processing for visualizing virtual camera information will bedescribed in detail below.

The display image generation unit 305 generates display images to beproduced on the display unit 402 of the user terminal 4. Display imagesgenerated in this case include a virtual viewpoint image generated bythe virtual viewpoint image generation unit 301 and information based ona result of the generation of virtual camera visualization informationgenerated by the visualization information generation unit 304.

The display image generation unit 305 may generate the virtual viewpointimages generated by the virtual viewpoint image generation unit 301 andthe virtual camera visualization information generated by thevisualization information generation unit 304 as data which can bedisplayed on the user terminal 4 as it is. The display image generationunit 305 may also generate a new image by overlapping these pieces ofinformation. The display image generation unit 305 may also generate adisplay image by adding further information to the virtual viewpointimages generated by the virtual viewpoint image generation unit 301, anda result of visualizing the virtual camera information generated by thevisualization information generation unit 304. Examples of theinformation to be added by the display image generation unit 305 mayinclude text data indicating the name of a player of an imaging targetsporting event and graphic data indicating the locus of a ball. Examplesof the information to be added by the display image generation unit 305may also include various information such as interface-displayed datafor receiving a predetermined input from the user. The display imagegeneration unit 305 generates a display image in the final state to betransmitted to the user terminal 4.

The management unit 306 receives user information including the useridentifier (ID) for identifying each user, account information relatedto each user, and attribute information of each user from the userterminal 4 via the communication unit 307 and stores the userinformation. The user information may be acquired from an apparatusdifferent from the user terminal 4. The management unit 306 may manageimages and various information including the virtual viewpointinformation for camera path setting transferred between the imageprocessing apparatus 3 and the user terminal 4, in association with theuser information. The communication unit 307 transmits and receivesvarious information including image, sound, and text data andinstructions for virtual camera path setting, to/from the user terminal4 via a network (not illustrated).

A configuration of the user terminal 4 will be described below. The userterminal 4 includes a terminal communication unit 401, a display unit402, a virtual camera path setting unit 403, and a transmission unit404. The terminal communication unit 401 transmits and receives variousinformation to/from the image processing apparatus 3. The display unit402 displays various information including information received from theimage processing apparatus 3. The virtual camera path setting unit 403receives an instruction for a virtual camera path from the user andtransmits the virtual viewpoint information for setting a virtual camerapath based on the instruction to the image processing apparatus 3 viathe terminal communication unit 401. In addition, the virtual camerapath setting unit 403 may be configured to, upon reception of aninstruction such as “view a portion where a target event is occurring”,calculate and acquire the position coordinates and light-of-sightdirection related to a virtual viewpoint image based on the instruction.The virtual camera path setting unit 403 may also be configured to, uponreception of an instruction such as “view a virtual viewpoint imagetargeting a specific player”, calculate and acquire the positioncoordinates and light-of-sight direction related to a virtual viewpointimage based on the instruction. The virtual camera path setting unit 403may also be configured to, upon reception of an instruction such as“view a fixed range of a ball”, calculate and acquire the positioncoordinates and light-of-sight direction related to a virtual viewpointimage based on the instruction.

The transmission unit 404 transmits user information including the userID for identifying each user, account information related to each user,and attribute information of each user to the image processing apparatus3 via the terminal communication unit 401.

A configuration of the image processing system 100 illustrated in FIG. 1is only an example. The image processing system 100 may have anotherapparatus having a part of the above-described components. The imageprocessing apparatus 3 may be composed of a plurality of apparatuses.

Operations of the image processing system 100 having the above-describedconfiguration will be described below. First, operations of the imageprocessing apparatus 3 will be described below with reference to theflowchart illustrated in FIG. 12. Processing in the flowchartillustrated in FIG. 12 is started when the image processing apparatus 3receives information from the user terminal 4. Processing in flowcharts(described below) is implemented when the CPU 1201 of the imageprocessing apparatus 3 executes a program to control informationcalculations and each hardware component. In addition, at least a partof steps in flowcharts (described below) may be executed by a dedicatedhardware component. The dedicated hardware component is, for example, anASIC, FPGA, and GPU.

In step S1301, the image processing apparatus 3 determines whetherinformation received from the user terminal 4 is the virtual viewpointinformation for identifying the position and orientation of a virtualcamera. When the image processing apparatus 3 does not determine thatthe information received from the user terminal 4 is the virtualviewpoint information (NO in step S1301), then in step S1302, the imageprocessing apparatus 3 determines whether the information received fromthe user terminal 4 is a request for acquiring the virtual cameravisualization information. When the image processing apparatus 3 doesnot determine that the information received from the user terminal 4 isa request for acquiring the virtual camera visualization information (NOin step S1302), the image processing apparatus 3 ends the processing.The image processing apparatus 3 may also be configured to performprocessing according to the acquired information.

On the other hand, when the image processing apparatus 3 determines thatthe information received from the user terminal 4 is the virtualviewpoint information (YES in step S1301), then in step S1303, the imageprocessing apparatus 3 sets the virtual camera path parameters based onthe received virtual viewpoint information. The image processingapparatus 3 sets imaging time related to the virtual viewpoint image tobe generated, and the viewpoint position and/or line-of-sight directionto be associated with the images. The image processing apparatus 3 mayset temporally continuous virtual camera path parameters and setparameters related to the viewpoint position and/or line-of-sightdirection related to the virtual viewpoint image as a still image.

In step S1304, the image processing apparatus 3 generates a virtualviewpoint image according to the set virtual camera path parameters. Instep S1304, as described above, the image processing apparatus 3generates a three-dimensional model by using images captured by aplurality of cameras and performs processing such as rendering on themodel to generate a virtual viewpoint image. In step S1305, the imageprocessing apparatus 3 transmits the generated virtual viewpoint imageto the user terminal 4.

On the other hand, when the image processing apparatus 3 determines thatthe information received from the user terminal 4 is a request foracquiring the virtual camera visualization information (YES in stepS1302), the processing proceeds to step S1306. The processing in stepS1306 will be described in detail below. In step S1307, the imageprocessing apparatus 3 transmits the generated virtual cameravisualization information to the user terminal 4.

Operations of the user terminal 4 will be described below with referenceto the flowchart illustrated in FIG. 13. Processing in the flowchartillustrated in FIG. 13 is started when the user terminal 4 receives aninstructing operation from the user. Processing in flowcharts (describedbelow) is implemented when the CPU 1201 of the user terminal 4 executesa program to control information calculations and each hardwarecomponent. In addition, at least a part of steps in flowcharts(described below) may be executed by a dedicated hardware component. Thededicated hardware component is, for example, an ASIC, FPGA, and GPU.

In step S1401, the user terminal 4 determines whether the instructingoperation received from the user is an instruction for setting a virtualcamera path. When the user terminal 4 does not determine that theinstructing operation received from the user is an instruction forsetting a virtual camera path (NO in step S1401), then in step S1405,the user terminal 4 determines whether the instructing operationreceived from the user is an instruction for displaying the virtualcamera visualization information. When the user terminal 4 does notdetermine that the instructing operation received from the user is aninstruction for displaying the virtual camera visualization information(NO in step S1405), the user terminal 4 ends the processing. The userterminal 4 may also be configured to perform processing according to thereceived instructing operation.

On the other hand, when the user terminal 4 determines that theinstructing operation received from the user is an instruction forsetting a virtual camera path (YES in step S1401), then in step S1402,the user terminal 4 transmits to the image processing apparatus 3 thevirtual viewpoint information for identifying the virtual camera pathbased on an instructing operation received from the user. In step S1403,the user terminal 4 determines whether a virtual viewpoint image basedon the transmitted virtual viewpoint information is received from theimage processing apparatus 3. When the user terminal 4 determines that avirtual viewpoint image is received from the image processing apparatus3 (YES in step S1403), then in step S1404, the user terminal 4 displaysthe received virtual viewpoint image.

On the other hand, when the user terminal 4 determines that theinstructing operation received from the user is an instruction fordisplaying the virtual camera visualization information (YES in stepS1405), then in step S1406, the user terminal 4 transmits a request foracquiring the virtual camera visualization information to the imageprocessing apparatus 3. After transmitting a request for acquiring thevirtual camera visualization information, then in step S1407, the userterminal 4 determines whether the virtual camera visualizationinformation is received from the image processing apparatus 3. When theuser terminal 4 determines that the virtual camera visualizationinformation is received from the image processing apparatus 3 (YES instep S1407), then in step S1408, the user terminal 4 displays thereceived virtual camera visualization information.

(Processing for Generating Virtual Camera Visualization Information)

The processing for generating the virtual camera visualizationinformation performed by the visualization information generation unit304 in step S1306 will be described below with reference to a specificexample. FIG. 2 illustrates an example of the virtual cameravisualization information generated by the visualization informationgeneration unit 304. FIGS. 2 and 3 illustrate examples of the virtualcamera visualization information showing the position and field of viewof virtual cameras C1 to C4 (Cu where u=1 to 4) separately set by fourdifferent users (user ID=u=1 to 4) in a scene including an imagingtarget. Referring to FIGS. 2 and 3, the virtual camera visualizationinformation displays computer graphics (CG) representing the field andstadium as imaging target areas and objects representing the virtualcameras C1 to C4 at corresponding positions on the virtual viewpointimage. Referring to FIGS. 2 and 3, an area A indicates an area forgenerating target object visualization information (described below).When the imaging target is a ball game such as soccer, the area A may beset as a three-dimensional space having a height of the range where aball can fly up from the field surface. The area A is divided into apredetermined number of blocks in each of the X, Y, and Z directions inthe three-dimensional coordinates. In the following descriptions, adivision block refers to each of blocks formed by dividing the area A,Information B indicates a range where virtual cameras can be set.Foreground objects P, Q, R, 5, and T indicate players and a ball.

The virtual camera visualization information illustrated in FIGS. 2 and3 may be generated by superimposing an object indicating a virtualcamera on a virtual viewpoint image. The position of a viewpoint relatedto the virtual viewpoint image on which the virtual camera visualizationinformation is to be superimposed may be arbitrarily changed on the userterminal 4. The image processing apparatus 3 may also determine theposition of a viewpoint of a virtual viewpoint image included in thevirtual camera visualization information based on the position of aviewpoint related to a virtual viewpoint image displayed by the userterminal 4. For example, the image processing apparatus 3 may match theline-of-sight direction of a virtual viewpoint image displayed by theuser terminal 4 with the light-of-sight direction of a virtual viewpointimage included in the virtual camera visualization information. Thismakes it easier to grasp the positional relationship between the virtualviewpoint image displayed by the user terminal 4 and the virtual cameravisualization information. The image processing apparatus 3 may set theposition of a viewpoint of a virtual viewpoint image included in thevirtual camera visualization information to a position on the fieldseparated from the position of a viewpoint related to a virtualviewpoint image displayed by the user terminal 4. This setting allowsthe user to look down on the positions of virtual cameras of other usersto grasp camera positions.

FIG. 3 illustrates the virtual camera visualization informationillustrated in FIG. 2 which is looked down on from above. As illustratedin FIGS. 2 and 3, to allow the user to look down on the positions ofvirtual cameras to grasp camera positions, it may be possible to set asa viewpoint position a range wider than the range where virtual camerapositions can be set in the virtual camera visualization information.

The display form of the virtual camera visualization information may bearbitrarily selected from a plurality of display forms including theexamples illustrated in FIGS. 2 and 3. The user terminal 4 may alsosimultaneously display as the virtual camera visualization information aplurality of display forms including the examples illustrated in FIGS. 2and 3. The user terminal 4 may also display the positions of the virtualcameras C1 to C4 by using camera-like objects.

The virtual camera visualization information illustrated in FIGS. 2 and3 is only as examples. The virtual camera visualization information mayinclude other information or may not include part of the informationillustrated in FIGS. 2 and 3. For example, in association with a virtualcamera, the virtual camera visualization information may include theuser ID as identification information for identifying the user who setthe virtual camera. The virtual camera visualization informationdisplayed on the user terminal 4 may not include information indicatingthe area A or B. The user terminal 4 may display or hide part of thevirtual camera visualization information illustrated in FIGS. 2 and 3 inresponse to a user's operation. The user terminal may display or hide,for example, information indicating the fields of view of the virtualcameras C1 to C4 in response to a user's operation. The user terminal 4may display the virtual camera visualization information illustrated inFIGS. 2 and 3 together with the virtual viewpoint image.

The target object visualization information as an example of the virtualcamera visualization information will be described below. The targetobject visualization information according to the present exemplaryembodiment is an image displayed on the display unit of the userterminal 4 used by a certain user, i.e., an example of informationindicating target objects (virtual viewing targets) included in thefields of view of virtual viewpoints specified by other users, in anidentifiable manner. A flow of processing for generating the targetobject visualization information will be described below with referenceto the flowchart illustrated in FIG. 4.

In step S1001, the image processing apparatus 3 acquires from thestorage unit 303 the foreground coordinates indicating the coordinatesof the foreground object included in the field of view of the virtualcamera C1. In step S1002, the image processing apparatus 3 determineswhether a division block includes the foreground coordinates. When theimage processing apparatus 3 determines that the division block includesthe foreground coordinates (YES in step S1002), then in step S1003, theimage processing apparatus 3 increments the count N (initial value is 0)supplied to the division block by 1. In step S1005, the image processingapparatus 3 increments the variable for identifying a virtual camera.Further, the image processing apparatus 3 performs the count processingin step S1003 for the following virtual camera. In step S1004, the imageprocessing apparatus 3 continues the processing in step S1003 until thecount processing is completed for all of the virtual cameras. FIGS. 5Aand 5B illustrate examples of results of the count processing in thescenes illustrated in FIGS. 2 and 3, respectively. FIG. 5A illustratesinformation indicating the count value simplified from the top surfaceside. This count value indicates the degree of attention of a divisionblock as an object targeted by each virtual camera.

When the image processing apparatus 3 determines that the countprocessing is completed for all of the virtual cameras (YES in stepS1004), then in step S1006, the image processing apparatus 3 determinesthe display color of each division block corresponding to the count. Theimage processing apparatus 3 may determine the display color of eachdivision block in the same procedure as a color heat map. The imageprocessing apparatus 3 may determine the display color according to apredetermined classification rule: red for the division block having thelargest count, blue for the division block having the smallest count,and orange, yellow, and green for division blocks having intermediatecounts in this order from the red side, for example. These colors areonly examples, and different colors may be used. The count N=0 maycorrespond to colorless, and the count may be represented by differentshades of a single hue or different textures. In addition, to smooth theboundaries between colors if the display colors of all division blocksare determined, the image processing apparatus 3 may perform boundaryprocessing such that rectangular boundary lines of division blocks areignored. The image processing apparatus 3 may also perform the boundaryprocessing such that boundaries match with contour lines of theforeground object in the division blocks.

The image processing apparatus 3 may display in the division block asthe target object visualization information the count number innumerical text form as it is. The image processing apparatus 3 may alsodisplay a moving image graphic in each division block as the targetobject visualization information and may indicate the count number bythe intensity of motion. In addition, a plurality of the representationmethods may be combined. The image processing apparatus 3 may make thetarget object visualization information translucent to maintain thevisibility of other information. In the above descriptions, the countnumber in division blocks included in the entire field of view of avirtual camera is increased in step S1001. However, processing is notlimited thereto. The count number in division blocks included in a partof the field of view of a virtual camera (for example, a part includingthe center of the field of view) be increased. Further, how much thecount of division blocks is increased may be determined according to apart of the field of view of the virtual camera in which the divisionblock is included.

An example display of the target object visualization information isillustrated in FIG. 5B. Referring to FIG. 5B, the image processingapparatus 3 represents the count by the density of texture to smoothlycorrect the boundaries between textures. In the above-described example,the image processing apparatus 3 displays the target objectvisualization information for emphasizing objects projected onto thevirtual viewpoint image according to a user-set viewpoint by usingcolors and moving images. In addition, in the above-described example,the image processing apparatus 3 displays colors and moving imagesaccording to the number of users who set virtual viewpoint images ontowhich objects are projected. Displaying such target object visualizationinformation allows the user to easily recognize the portion to which alarger number of virtual cameras are oriented and the object on whicheach user focuses attention. An example display of the target objectvisualization information is not limited thereto. For example, icons andtext information may be used. Although, in the above-described example,information indicating division blocks in an identifiable manner astarget objects included in the field of view of virtual cameras aregenerated, target objects are not limited to division blocks. Forexample, the image processing apparatus 3 may determine which person isincluded in the field of view of each virtual camera and generateinformation indicating a person in an identifiable manner as a targetobject included in the field of view of the virtual camera. For example,the image processing apparatus 3 may determine which portion of a personis included in the field of view of each virtual camera and generateinformation indicating the portion of the person in an identifiablemanner as a target object included in the field of view of the virtualcamera.

(Recommendation Processing Based on Virtual Camera VisualizationInformation)

A flow of recommendation processing based on the virtual cameravisualization information will be described below with reference to theflowchart illustrated in FIG. 7. The recommendation processing based onthe virtual camera visualization information is processing for notifyinga user who is viewing an area different from an area being viewed bymany users, of the area captured by a larger number of other virtualcameras as recommendation information. The recommendation processing isalso processing for moving, upon reception of a request from the user, avirtual camera to the area captured by a larger number of other virtualcameras, i.e., the area on which many users focus attention.

In step S1007, the image processing apparatus 3 compares the viewingnumber (N1max corresponding to the count value in the processingillustrated in FIG. 4) of all users of the foreground object within theangle of view of the virtual camera of the user subjected to processing,with the maximum count (Nmax) in the entire area A. When the result ofthe comparison in step S1007 is N1max<<Nmax (YES in step S1008), then instep S1009, the image processing apparatus 3 generates a recommend imageand transmits the generated recommend image to the user terminal 4. Acriterion for determining whether N1max is sufficiently smaller thanNmax is arbitrarily predetermined. For example, the image processingapparatus 3 determines whether N1max is equal to or smaller than a fixedratio of Nmax. Since the number of users is 4, in the exampleillustrated in FIG. 2, the following description will be made on thepremise that N1max<<Nmax although the difference between N1max and Nmaxis small.

The user terminal 4 displays the recommend image received from the imageprocessing apparatus 3. The recommend image is an example of informationindicating in an identifiable manner a target object included within thefield of view of a virtual camera specified by a user who is differentfrom the user of the user terminal 4. An example display of a recommendimage will be described below with reference to FIGS. 6A to 6D. FIG. 6Aillustrates an example screen of the display unit 402 of the userterminal 4 a which displays a virtual viewpoint image. Referring to FIG.6A, the screen displays a virtual viewpoint image (an image of thevirtual camera C1) based on a virtual camera path specified by the user1 of the user terminal 4 a. When the image processing apparatus 3performs the processing in step S1007 on the user terminal 4 adisplaying the virtual viewpoint image illustrated in FIG. 6A, the imageprocessing apparatus 3 determines that N1max<<Nmax. More specifically,the image processing apparatus 3 determines that the number of all theusers viewing the objects projected onto the virtual viewpoint imagedisplayed on the user terminal 4 a is extremely smaller than the viewingnumber of the object shot by the largest number of other user terminals.Accordingly, the image processing apparatus 3 generates the recommendimage for the user terminal 4 a and transmits the recommend image to theuser terminal 4 a. FIG. 6B illustrates an example of the recommend imagegenerated by the display image generation unit 305 of the imageprocessing apparatus 3 and displayed on the user terminal 4. Referringto FIG. 6B, the recommend image is superimposed on the virtual viewpointimage displayed on the user terminal 4 a, Referring to FIG. 6B, agraphic indicating the count number of the area A and a title text“recommend” are displayed in an area at the top right of the virtualviewpoint image. The user terminal 4 may display the recommend image ina blinking state to draw the attention of the user 1. In this case, therecommend image may be displayed as a three-dimensional perspective viewillustrating the contour of the area A with a wire frame. Theperspective direction of the recommend image may be close to theline-of-sight direction of the virtual camera C1 to make it easier forthe user 1 to grasp the positional relation with the virtual camera C1currently being viewed by the user 1. In addition, the recommend imagemay be the virtual viewpoint image viewed from the perspective directionwith information superimposed thereon.

In step S1010, the image processing apparatus 3 determines whether theuser terminal 4 changes the virtual camera position based on therecommend image. The image processing apparatus 3 determines whether arequest for changing the virtual camera position based on the recommendimage is received from the user terminal 4. When a tap, selectiondetermination, or other predetermined instructing operation is performedon the recommend image, the user terminal 4 transmits a request forchanging the virtual camera position based on the recommend image to theimage processing apparatus 3.

When the image processing apparatus 3 determines that the user terminal4 changes the virtual camera position based on the recommend image (YESin step S1010), the visualization information generation unit 304 of theimage processing apparatus 3 acquires the current virtual cameraposition of the user terminal 4 which is a processing target. In stepS1011, the image processing apparatus 3 render the informationindicating the current virtual camera position on the recommend image.To allow the user 1 to recognize the current virtual camera position,the image processing apparatus 3 generates rendering information whichshows the position of the viewpoint of the virtual camera C1. Referringto FIGS. 6A to 6D, based on the rendering information from the imageprocessing apparatus 3, the user terminal 4 a blinks the foregroundobject captured by the virtual camera C1 together with a text “YourView” as information indicating the position of the viewpoint of thevirtual camera C1. In step S1012, the image processing apparatus 3transmits information for enlarging the recommend image to the userterminal 4.

FIG. 6C illustrates an example display of an enlarged recommend image onthe user terminal 4 a. Referring to FIG. 6C, the foreground objectcaptured by the virtual camera C1 (information indicating the positionof the viewpoint of the virtual camera C1) is represented by a stripedpattern. The information indicating the position of the viewpoint of thevirtual camera C1 may be displayed in a blinking state, or in a statewhere the color of the foreground object captured by the virtual cameraC1 is changed or the foreground object captured by the virtual camera C1is surrounded by a circular or rectangular curve. The informationindicating the position of the viewpoint of the virtual camera C1 may beinformation showing the current position of the virtual camera C1 indiverse ways, for example, by displaying a graphic such as a circle andcamera icon at the position of the virtual camera C1. The imagedisplayed on the user terminal 4 a may be provided with objectidentification information. Examples of the object identificationinformation include the name of the person corresponding to theforeground object captured by the virtual camera C1, and the name of theperson corresponding to the target object included in the field of viewof other virtual cameras. The object identification information is notlimited to a person's name and may be, for example, the ID assigned to aperson and the name of a target portion of a person.

In step S1013, the image processing apparatus 3 determines whether aninstruction for changing the virtual camera path is received from theuser terminal 4. When a tap, selection determination, or otherpredetermined instructing operation is performed on an area with a largecount number in the recommend image, the user terminal 4 transmits aninstruction for changing the virtual camera path to the image processingapparatus 3. When an instruction for changing the virtual camera path isreceived (YES in step S1013), then in step S1014, the acquisition unit302 of the image processing apparatus 3 automatically calculates avirtual camera path along which the virtual camera C1 is moved up to theposition where a selected area is shot, and acquires the parameters ofthe virtual camera path. The image processing apparatus 3 mayautomatically calculate a virtual camera path so as to translate thevirtual camera C1 without changing a zoom factor of the virtual camera.In step S1014, the image processing apparatus 3 may calculate a virtualcamera path along which the virtual camera C1 is moved from the currentposition to the recommend position. The movement of the virtual cameraC1 is shot in a virtual viewpoint image to make it easier for the userto keep correct recognition of the virtual camera position. The imageprocessing apparatus 3 may also calculate a virtual camera path alongwhich a selected area is shot while rotating the line-of-sight directionwith the current virtual camera position fixed. The image processingapparatus 3 may also calculate a virtual camera path along which,instead of moving the virtual camera from the current position to theposition where the selected area is captured, the virtual cameraposition is switched from the current position to that position. Theuser terminal 4 may also be configured to automatically calculate avirtual camera path. The image processing apparatus 3 or the userterminal 4 may select a virtual camera specified in the virtual cameravisualization information and determine the position of the selectedvirtual camera as a viewpoint for a virtual viewpoint image to begenerated.

In step S1015, the image processing apparatus 3 transmits an instructionfor ending display of the recommend image to the user terminal 4. Theuser terminal 4 ends display of the recommend image. In step S1016, theimage processing apparatus 3 generates a virtual viewpoint imageaccording to the virtual camera path calculated in step S1014, and thevirtual camera is moved. FIG. 6D illustrates an example of a virtualviewpoint image to be displayed on the user terminal 4 after moving thevirtual camera based on the recommend image.

Through the above-described recommendation processing based on thevirtual camera visualization information, the user comes to know theexistence of an area on which many other users focus attention. If theuser wants to view the area, he can orient or move a virtual camera tothe area. In the examples illustrated in FIGS. 6A to 6D, while the userselects an area with a large count, the user may select an area with asmall count or the like to arbitrarily select an area to which a virtualcamera is moved. Although, in the processing illustrated in FIG. 7, therecommend image is automatically displayed when N1max<<Nmax, therecommend image may be automatically displayed in response to aninstructing operation performed by the user at arbitrary timing.

A modification of the virtual camera visualization information will bedescribed below. A modification of processing for generating the virtualcamera visualization information will be described below with referenceto the flowchart illustrated in FIG. 8. In the processing illustrated inFIG. 8, the image processing apparatus 3 generates visualizationinformation showing each virtual camera as an arrow or quadrangularpyramid.

In step S3001, the visualization information generation unit 304 of theimage processing apparatus 3 acquires information about the position,line-of-sight direction, and angle of view of a virtual camera Cu fromthe storage unit 303. In step S3002, the visualization informationgeneration unit 304 of the image processing apparatus 3 changes theinformation of the virtual camera Cu to an arrow or quadrangularpyramid. When changing the information indicating the virtual camera toan arrow, the image processing apparatus 3 sets the position coordinatesof the virtual camera Cu as the starting point coordinates of the arrow,and sets the line-of-sight direction of the virtual camera Cu as theorientation of the arrow. The length of the arrow from the startingpoint is a fixed length. FIG. 9 illustrates an example where an arrow isused as information for visualizing a virtual camera. Referring to FIG.9, arrows are rendered not as lines but in three-dimensional form. Inthe example illustrated in FIG. 9, the area A as a range where virtualcamera information is visualized coincides with an area B (a range wherevirtual cameras can be set).

When information indicating a virtual camera is represented as thequadrangular pyramid, the image processing apparatus 3 sets the positioncoordinate of the virtual camera Cu as the vertex coordinates, sets theline-of-sight direction of the virtual camera Cu as the orientation fromthe vertex to the central point of the base end surface, and sets thehorizontal and vertical angles of view of the virtual camera Cu as theangles between two pairs of facing triangular lateral surfaces,respectively. The distance from the vertex to the base end surface ofthe quadrangular pyramid is a fixed length from the vertex. FIG. 10illustrates an example where a quadrangular pyramid is used asinformation for visualizing a virtual camera. FIG. 10 illustrates allridgelines of each quadrangular pyramid. However, to represent a statewhere the field of view spreads to infinity, the quadrangular pyramidmay be illustrated in gradation starting from the vertex so that thebase end surface disappears. Further, the quadrangular pyramid mayextend up to the boundary of the area A.

In addition to the information illustrated in FIG. 9 or 10, theinformation generated in step S1006 may be added to the foregroundobject. The information indicating a virtual camera is not limited tothe above-described example. Other information including other icons,text information, and symbols may be used.

According to the above-described exemplary embodiment, it is possible todirectly show the position where each virtual camera exists, thedirection to which the virtual camera is oriented, the type (wide-angleor zoom) of the virtual camera, and the angle of view of the virtualcamera. In addition, the virtual camera visualization information may berepresented by a combination of arbitrary visualization techniquesincluding a count number in graphical form and an arrow or quadrangularpyramid representing a virtual camera itself. For example, by displayingthe count in graphical form and the quadrangular pyramid, the user comesto know where a target area is, the direction from which the area isfocused, and the angle of view with which the area is focused.

When generating the virtual camera visualization information, thevisualization information generation unit 304 may categorize users basedon the user information acquired from the management unit 306 andgenerate visualization information for each user category. Examples ofuser categories are the age, gender, hometown, and current residentaddress. Moreover, various categories are conceivable such as theexperience data and favorite team about a specific sport, and experiencedata on virtual camera operations. When displaying a visualizationresult for each user category, display may be simply changed for eachcategory. Alternatively, when simultaneously displaying all categories,display may be differentiated by color and texture. Further, the usercategory name itself may be displayed as a text near the informationindicating a virtual camera. More specifically, the visualizationinformation generation unit 304 does not need to generate informationindicating the virtual camera positions for all users stored in thestorage unit 303. The visualization information generation unit 304 maygenerate the virtual camera visualization information related to oneuser or a part of users.

Although, in the above-described exemplary embodiment, the virtualcamera information of a plurality of users simultaneously existing at acertain time is visualized, the visualization information generationunit 304 may generate and display, for example, the virtual cameravisualization information indicating an area which has been often viewedby a user through entire one game. The virtual camera information may beaccumulated in the successive temporal direction and visualized also fora plurality of users. In this case, in the above-describedrecommendation processing, the user terminal 4 a may display the virtualcamera visualization information related to a second virtual viewpointpreviously set by the user when, for example, the user terminal 4 a isdisplaying a first virtual viewpoint image based on a first virtualviewpoint set by a user a of the user terminal 4 a.

According to the above-described exemplary embodiment, it is possible tocalculate and graphically display a count number and show a virtualcamera as an arrow or quadrangular pyramid, which makes it easier forthe user to grasp an area on which other users focus attention. Inaddition, the user is able to move a virtual camera to an arbitraryposition while referring to the positions of virtual cameras of otherusers. The user is also able to move his or her own virtual camera toone of the positions of virtual cameras of other users.

While, in the above descriptions, the image processing apparatus 3 isconfigured to generate the virtual camera visualization information, theinformation may be generated by the user terminal 4. In this case, theimage processing apparatus 3 may transmit information such as thecoordinates information of the virtual cameras stored in the storageunit 303 to the user terminal 4, and the user terminal 4 may generatethe virtual camera visualization information based on the receivedinformation.

While, in the above-described recommendation processing, an area shot bya larger number of other virtual cameras is recommended to a user who isviewing an area different from areas being viewed by many users, therecommendation processing is not limited thereto. For example, the userterminal 4 may display the virtual camera visualization information fornotifying a user who is viewing the same area as other users that otherusers are also shooting the same area by using virtual cameras. Morespecifically, the image processing apparatus 3 may notify the userterminal 4 a displaying the first virtual viewpoint image based on thefirst virtual viewpoint set by the user a, of the virtual cameravisualization information related to the second virtual viewpoint (thesame position as the first virtual viewpoint) set by the user b in theuser terminal 4 b.

While the present disclosure has specifically been described in detailbased on the above-described exemplary embodiment, the presentdisclosure is not limited thereto but can be modified in diverse wayswithin the ambit of the appended claims. The above-described exemplaryembodiment, for example, allows a user to easily recognize informationabout a virtual viewpoint of a virtual viewpoint image currently beingviewed by other users.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Btu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, the scope of the following claims are to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2017-206013, filed Oct. 25, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus, comprising:one or more hardware processors; and one or more memories which storeinstructions executable by the one or more hardware processors to causethe information processing apparatus to perform at least: specifying afirst virtual viewpoint corresponding to a first virtual viewpoint imageto be generated based on captured images of an imaging target areacaptured by a plurality of imaging apparatuses at different positions;specifying a target object included in a field of view of a secondvirtual viewpoint corresponding to a second virtual viewpoint image tobe generated based on the captured images; and causing a displayapparatus to display the first virtual viewpoint image corresponding tothe specified first virtual viewpoint with information for specifyingwhether an object in the first virtual viewpoint image is the specifiedtarget object included in the field of view of the second virtualviewpoint and for specifying at least one of a position of the secondvirtual viewpoint, a direction of view from the second virtual viewpointand an angle of view of the second virtual viewpoint.
 2. The informationprocessing apparatus according to claim 1, wherein the display apparatusis caused to display the target object with a color distinguishable fromcolors of other objects.
 3. The information processing apparatusaccording to claim 1, wherein the display apparatus is caused to displaythe target object by a moving image such that the target object isdistinguishable from other objects.
 4. The information processingapparatus according to claim 1, wherein the display apparatus is causedto display information indicating a degree of attention according to thenumber of virtual viewpoints which include the target object inrespective fields of view.
 5. The information processing apparatusaccording to claim 4, wherein the display apparatus is caused to displaythe target object by using at least one of a color according to thedegree of attention and a moving image according to the degree ofattention.
 6. The information processing apparatus according to claim 1,wherein the display apparatus is caused to display informationindicating an area where the target object is positioned.
 7. Theinformation processing apparatus according to claim 1, wherein thedisplay apparatus is caused to display information for identifying thetarget object.
 8. The information processing apparatus according toclaim 1, wherein the display apparatus is caused to display theinformation on the first virtual viewpoint image.
 9. The informationprocessing apparatus according to claim 1, wherein an object included ina predetermined part of the field of view of the second virtualviewpoint is specified as the target object.
 10. The informationprocessing apparatus according to claim 9, wherein the predeterminedpart of the field of view of the second virtual viewpoint includes acenter of the field of view of the second virtual viewpoint.
 11. Theinformation processing apparatus according to claim 1, wherein a userdesignating the first virtual viewpoint is different from a userdesignating the second virtual viewpoint.
 12. The information processingapparatus according to claim 1, wherein an object included in each offields of view of a plurality of virtual viewpoints different from thefirst virtual viewpoint is specified as the target object.
 13. Theinformation processing apparatus according to claim 1, wherein thedisplay apparatus is further caused to display the second virtualviewpoint image in response to an operation on an image including theinformation displayed by the display apparatus.
 14. The informationprocessing apparatus according to claim 1, wherein the target object isa person or a part of a person positioned in the imaging target area.15. An information processing method comprising: specifying a firstvirtual viewpoint corresponding to a first virtual viewpoint image to begenerated based on captured images of an imaging target area captured bya plurality of imaging apparatuses at different positions; specifying atarget object included in a field of view of a second virtual viewpointcorresponding to a second virtual viewpoint image to be generated basedon the captured images; and causing a display apparatus to display thefirst virtual viewpoint image corresponding to the specified firstvirtual viewpoint with information for specifying whether an object inthe first virtual viewpoint image is the specified target objectincluded in the field of view of the second virtual viewpoint and forspecifying at least one of a position of the second virtual viewpoint, adirection of view from the second virtual viewpoint and an angle of viewof the second virtual viewpoint.
 16. The information processing methodaccording to claim 15, wherein, the display apparatus is caused todisplay the target object with at least one of a color distinguishablefrom colors of other objects.
 17. The information processing methodaccording to claim 15, wherein, the display apparatus is caused todisplay information indicating a degree of attention according to thenumber of virtual viewpoints which include the target object inrespective fields of view is displayed.
 18. A non-transitory storagemedium storing a program for causing a computer to execute aninformation processing method, the information processing methodcomprising: specifying a first virtual viewpoint corresponding to afirst virtual viewpoint image to be generated based on captured imagesof an imaging target area captured by a plurality of imaging apparatusesat different positions; specifying a target object included in a fieldof view of a second virtual viewpoint corresponding to a second virtualviewpoint image to be generated based on the captured images; andcausing a display apparatus- to display the first virtual viewpointimage corresponding to the specified first virtual viewpoint withinformation for specifying whether an, object in the first virtualviewpoint image is the specified target object included in the field ofview of the second virtual viewpoint and for specifying at least one ofa position of the second virtual viewpoint, a direction of view from thesecond virtual viewpoint and an angle of view of the second virtualviewpoint.
 19. The information processing apparatus according to claim1, wherein the display apparatus is caused to display first informationindicating whether an object in the first virtual viewpoint image is thespecified target object included in the field of view of the secondvirtual viewpoint and second information, which is different from thefirst information, indicating at least one of a position of the secondvirtual viewpoint, a direction of view from the second virtual viewpointand an angle of view of the second virtual viewpoint.
 20. Theinformation processing apparatus according to claim 1, wherein thedisplay apparatus is caused to display one piece of informationindicating whether an object in the first virtual viewpoint image is thespecified target object included in the field of view of the secondvirtual viewpoint and indicating at least one of a position of thesecond virtual viewpoint, a direction of view from the second virtualviewpoint and an angle of view of the second virtual viewpoint.